Decoding Science: Lessons from the Leading Edge

Feature Story

Health and Medicine

Pandemics

Last update May, 12 2021

As COVID-19 spread, scientists sprang into action. Their early studies hold lessons for any new area of science: Approach with caution and stay open to change.

The spring of 2020 was a period many of us will remember for the rest of our lives. In a few short weeks, normalcy dissolved into catastrophe as a terrible virus swept through communities and across borders. It was a time of disruption, fear, and grief — a time of great uncertainty and endless questions.

What sort of virus was it? Did it travel through water, food, air? How did it sicken people? Where had it come from? Where was it now? What could be done to stop it?

Scientists around the globe pivoted to face the emerging threat, repurposing equipment and expertise to understand the virus and help inform critical decisions being made in hospitals, statehouses, board rooms, and homes.

Remarkably quickly, results started pouring in. Scientists shared insights into the virus, its health impacts, its spread, and more. The virus and the disease it causes soon gained names — SARS-CoV-2 and COVID-19, respectively — and the world gained a clearer picture of the pandemic we faced.

Yet despite their importance, the studies shared in early 2020 had some key weaknesses. It is likely that some researchers made mistakes that might have been avoidable with the luxury of time. Other weaknesses were more fundamental, inherent to emerging areas of science. For starters, almost all of the early studies were missing the element of controlled repetition.

First, fast ... flawed?

Repetition is a crucial part of science. Reproducibility — when research repeated with the exact same data and code yields the same results — helps verify that the analysis of the original study is valid. Replicability — when studies examine the same questions by using similar methods and collecting new data and come to similar conclusions — helps build confidence in a body of knowledge.

Reproducibility and replicability are both vital to scientific knowledge. Both take time.

An emerging area of science — a never-before-seen virus, for example — is by definition uncharted territory. Scientists often view early studies with skepticism not because the science is bad, but because there simply hasn’t been enough time to repeat the studies, confirm the findings, and build confidence in their conclusions. It takes repetition to build confidence in studies, and it takes many studies to build scientific consensus and knowledge.

There are many reasons why studies that are first or fast might not hold up to repeated testing. One is a lack of knowledge; until scientists know more about what they’re studying, it can be hard to ask the right questions or understand how the new area compares to established knowledge. Another is a lack of methods; sometimes new science requires new instruments or approaches.

Of course, scientists have plenty of existing knowledge and well-honed methods at their disposal — from studying other coronaviruses or other economic shocks, for example. But it can be difficult to tell which past lessons or tools apply under a new set of circumstances and what might be different this time. In short, it’s hard to know what we don’t know.

In the case of COVID-19, most early studies also lacked other elements that can help strengthen a study, such as controls that are unaffected by the phenomenon being studied. For instance, before there were reliable tests to detect the virus, it was impossible even to compare people who were exposed or infected to those who were not. This, in part, is why it was so challenging to pin down how the virus was transmitted or why some people recovered and others did not.

In many ways, it’s simply not possible (or ethical) to study a real-life pandemic, unfolding in real time, using the same methods you would use in the laboratory. Instead, scientists had to continually adapt their methods on the fly, in response to incremental insights gained along the way.

Recognizing gaps

As the first weeks of the pandemic turned into months, scientists and the public learned much more about COVID-19 and its repercussions for health and society. As some answers became clearer, the world’s focus turned to new questions: Would vaccines work? What treatments are best? When is it safe to reopen schools? How can we mitigate the pandemic’s impacts on mental health, vulnerable populations, and the economy?

Along with other organizations, the National Academies of Sciences, Engineering, and Medicine sought to provide science-based information to decision-makers and the public. A common theme pervades many of these efforts. While putting forth the best available information based on evidence gathered to date, scientists were also careful to recognize the knowledge gaps that remained.

This recognition of the limitations of emerging science is evident in a series of rapid expert consultations that weighed in on topics such as testing, social distancing, and masks in the early months of the pandemic. It’s also present in a study on reopening classrooms published in the summer of 2020, which offers a detailed analysis of the science on COVID-19 and children while noting gaps in the evidence on transmission by children, the role of schools in contributing to viral spread, the extent of airborne transmission, and the effectiveness of mitigation strategies.

Speakers at a workshop in the fall of 2020 on airborne transmission of SARS-CoV-2 recounted some of the ways understanding of the virus had shifted over time. For example, many scientists initially thought that people infected with SARS-CoV-2 were contagious only when exhibiting symptoms, but later studies confirmed that the virus could also be passed by people with no symptoms at all. Understanding of the role of tiny airborne particles, which behave differently from larger droplets such as those produced by coughs and sneezes, also evolved over time.

When developing treatments and vaccines, scientists perform multiple studies to build knowledge about their safety and effectiveness, both before and after they are available to the public. A report on allocation of COVID-19 vaccines notes the importance of communicating the uncertainties that are inherent in clinical trials and the possibility that recommendations may change as vaccines roll out in real-world settings.

Making decisions with imperfect data

Multiple studies almost always provide stronger evidence than any single study, but early studies can still help inform decision-making when interpreted appropriately. In the COVID-19 pandemic, governments and individuals have often had to make decisions with imperfect data.

It is helpful to understand the strengths and weaknesses of different types of information. A guide for decision-makers, published in June 2020, outlines seven types of data that could inform decisions related to the pandemic, along with their sources of bias and uncertainty. For example, reported deaths from COVID-19 can be affected by the accuracy of cause-of-death determinations and reflect the state of the outbreak several weeks previously, since deaths usually lag infections by several weeks. The fraction of viral tests that are positive, on the other hand, can offer a more timely assessment of an outbreak but can overestimate or underestimate disease prevalence based on the number of asymptomatic people who are tested relative to symptomatic people.

Systems for sharing scientific findings also provide context on the strength of the evidence. For example, findings from many studies during the COVID-19 pandemic have been made available in preprints — preliminary reports shared before a study has undergone careful review by experts in the field. In general, findings that have passed peer review are considered more reliable.

Whether dealing with a newly discovered virus or a newly discovered galaxy, repetition is critical to building scientific knowledge. To that end, it is valuable to incentivize, facilitate, and perform reproducibility and replicability studies for important work in all fields of science.

When the COVID-19 pandemic focused the world’s attention on science, scientists around the world began gathering data and sharing results with remarkable speed. Some of what we learned through those early studies has held up through repeated study. Some has not. As old questions are settled and new ones arise, rigorous science helps us become a little more certain — even in times of great uncertainty.